Strands released Strands Decider 2B, a 2-billion-parameter model designed to quickly pick between predefined options rather than generate arbitrary text. It runs locally in ~115ms and ranks third among 2B-class decision models on accuracy and calibration benchmarks.

• Decision models sacrifice flexibility for speed and reliability: they choose from given options and assign confidence scores, making them 100x faster than LLMs for simple classification tasks but unsuitable for text generation, coding, or reasoning.
• The 2B parameter size hits a sweet spot for local experimentation—small enough to run on consumer hardware, large enough to handle meaningful work like tool selection, routing, and guardrails in agentic workflows.
• Strands open-sourced the full model, training data, and 19 iterations of architecture refinements on GitHub and Hugging Face, explicitly inviting community contributions to the decision model space.

Kev is a family of small language models (0.8B to 9B parameters) built on Qwen3.5 that make yes/no, multiple-choice, and rating decisions on text inputs. You can run them locally on your own hardware with an API compatible with TypeSafe's System One, plus there's a web playground for testing.

• Three model sizes available with published accuracy metrics: Kev-9B reaches 85.2% accuracy on unseen datasets and trails the proprietary Jev model by only 3.5 points, while Kev-4B and Kev-0.8B offer smaller footprints for different use cases.
• All models run on consumer hardware (CUDA, ROCm, Apple Silicon) with the 4B and 9B versions fitting on a 32GB Mac in bf16 precision, plus you can train your own weights using provided code and evaluation data.
• Probabilities are calibrated by default and the models output confidence scores alongside answers, reducing confident errors to 4% on new data compared to 8.7% without calibration.

Someone built a multi-GPU homelab for local AI inference using a Framework Desktop, RTX 5090 eGPU, and two DGX Spark units, routing requests between local and cloud models depending on latency needs. They're currently at 60-70% local inference and plan to move toward 100%, though they admit the eGPU was a mistake and you don't need this much hardware to start.

• The Qwen 3.8 27B model on the 5090 eGPU hits 150+ tokens/second for fast inference, while the Deepseek v4 Flash models on the DGX Sparks handle slower but higher-quality batch processing and background jobs.
• A custom routing plugin (Arch-Router) decides whether each request goes to local models or cloud/frontier models based on latency requirements, currently keeping about 60-70% of workloads local.
• The hardware stack is overkill for most people—started with just a Mac mini and kept adding. The eGPU specifically isn't worth the investment.

Perplexity's Portable Computer lets Windows users run AI agents locally on NVIDIA RTX GPUs, keeping sensitive data on-device while handling multi-step tasks without consuming cloud credits. The tool connects to common apps like Outlook, Gmail, and GitHub, with an option to offload complex work to cloud models when needed.

• Runs on local NVIDIA GeForce RTX and RTX PRO GPUs with 24GB+ VRAM, requiring no manual model selection or complex software setup
• Handles real workflows: reviewing GitHub PRs, analyzing financial documents with exact file citations, identifying user drop-off points in product funnels
• Sensitive information stays on the device; users approve before any data leaves for cloud processing

Meta open-sourced Muse Glimmer, a 30-billion-parameter model designed to run on consumer GPUs and laptops for local AI agent tasks like scheduling, coding, and tool use. The model uses quantization and speculative decoding to fit within 20-32GB of memory while maintaining speed for real-time interaction. It's available now under Apache 2.0 license with integrations for llama.cpp, MLX, and other frameworks.

• Meta open-sourced Muse Glimmer, a 30B-parameter agentic model quantized to under 20GB so it runs locally on consumer GPUs/laptops (24-32GB) while competing with Gemma 4-31B and Qwen 3.6-27B.
• Speculative decoding with a lightweight drafter model proposes token blocks at once instead of one token at a time, speeding up reasoning and tool calls without changing outputs.
• Training combined logit distillation from a larger teacher model, agent-heavy mid-training data, and post-training RL/distillation across reasoning, coding, and agentic tasks.
• Released under Apache 2.0 with upcoming llama.cpp, MLX, and ExecuTorch integrations and support from Ollama, LM Studio, Together AI, and chipmakers like NVIDIA, Intel, AMD, and Arm.

This guide shows how to run Z.ai’s open-source GLM-5.2 model on local hardware using Unsloth Dynamic GGUF quantizations. It covers memory requirements for 1-bit and 2-bit setups, recommended inference settings, and step-by-step instructions for Unsloth Studio and llama.cpp. The article also explains KLD benchmarks and quantization accuracy trade-offs.

• GLM-5.2's 744B-parameter model can run on a 24GB GPU plus 256GB RAM using Unsloth's 2-bit dynamic quant (239GB, ~82% top-1 accuracy), an 84% size cut from the full 1.5TB model.
• The 1-bit quant shrinks total memory needs to ~223GB but drops accuracy to ~76.2%.
• 4-bit and 5-bit dynamic quants are nearly lossless per KLD testing, making them the better choice when accuracy on out-of-distribution tasks matters most.
• Dynamic quantization keeps critical layers at higher precision while aggressively compressing less important ones, which is why even 1-2 bit versions stay broadly usable despite the accuracy trade-off.

The author connects a 16 GB Mac Mini to a 64 GB MacBook Pro using LM Studio Link’s encrypted mesh VPN, offloading heavy model inference to the more powerful machine without exposing ports or tweaking firewalls. This setup lets you run large LLMs on low-RAM devices as if they were local, with no cloud or API key hassles.

• LM Studio Link uses Tailscale's WireGuard mesh VPN to let a 16GB Mac Mini offload inference to a 64GB M4 Max MacBook Pro with zero port forwarding or firewall config
• Remote inference performance is essentially native: 87 tokens/sec on GPT-OSS 20B, with a 1,139-token answer generated in half a second
• Free tier supports two users and up to ten devices, enabling use cases like thin-client Raspberry Pi setups or small teams sharing one powerful machine
• Main tradeoffs are unchanged initial model load times and total loss of access if the host machine goes offline